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ABSTRACT , . ^ u - • 

In order to evaluate on-the-gob physician 

performance, four different composite scores were derived from 
different weightings of 80 criteria. Weights were assigned both 
objectively, through statistical methods, and subjectively, based 
only upon personal judgment. Comparing these results with scores 
based upon appraisal of the individual by other physicians produced 
remarkably similar results, especially considering the instability of 
many of the criteria. (BH) 
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Abstract No. 42: Program : Seventh Aimijal Conferen ce on Research in 

Medical Education , 1968, pp . 30-31. 

The Association of American Medical Colleger 

Svntneses of Multiple Criteria of Physic ian Perform ance. C. W. Taylor, 

Ph.D., E. G . "Lew^, Ph.D. , D. E. Nelson, B.S. , 07*0. Loughmiller, M.A., 
and P. B. Price, M.D. , University of Utah. 

In 1964, Price, et al. ( J . Med . Educ , 39: 203-211, 1964) reported a 

medical criterion study in which 80 criteria of physician performance^ 
were collected and analyzed for a sample of several hundred Utah physicians. 
The purpose of this paper is to determine how well these same criteria 
can be combined or synthesized into alternate versions of a single, 
summary score of over-all physician performance. These surnnary scores 
provide new criteria for validating predictors. They will also serve 
as ’’landmarks” against which new criterion scores, soon to be collectaJ , 
will be compared to see to what degree they extend che range of 
physician performances already measured. 

Four different composite or summary scores for each of 205 specialists 
studies by Price, et al. , were derived by vzeightlng each of the 80 
criterion measures, in four different w3-ys. (s) C'-?iiiposrte scopes wore 
obtained by simply summing the 80 unweighted criterion scores for each 
physician. (b) The 80 criteria v?ere summed after : ach criterion had been 
assigned a weight based on its judged importance to medical practice. 

These weights were assigned to the SO criteria b’^ a panel of five physician 
judges. (c) Medical experts gave an over-all raring of 1 to 10 to each 
specialist on the basis of his 80 performance cr *\_ria contained in a 
coded, anonymous folder. (d) The weight for each critei'ion was derived 
by reversing the typical application of the statistical technique of 
multiple correlation, using the 80 performance variables as ’’predictors” 
and the over-all rating of performance (described in 3 above) as the 
criterion. 

After derivation of the composite ci^iterion scores, half of the 
intercorrelations among the four synthesized criterion scores were found 
to be over .85. 

We plan to obtain a fifth summary score by having a number* of know- 
ledgeable physicians make an over— all appraisal of each specialist by 
name. These ratings, obtained independently of the 80 criteria, will 
provide another indication of how w^-11 our syntheses of the 80 perform- 
ance measures reflect actual physician practice. Early indications are 
that our comx>osite scores hold much promise as reasonably accurate 
-criteria of on-the-job physician pei^formance for use until they can be 
further extended and refined. 
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In 1964, Price, et. al . (J- Hed. Educ., 39:203-211, 1964) reported 
a medical criterion study in which 80 criteria of physician perfoirnance 
were collected and analyzed for a sample of several hundred Utah Physicians 
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(see Table 1). In the first phase of the present study those 80 criteria 
were combined in various ways to produce four summary scores of overall 
physician performance. This procedure was deemed desirable for two reasons: 
(1) Such summary scores would provide additional criteria for validating 
predictors; (2) the summary scores could serve as "landmarks" against 
which, new criterion scores, soon to be col..ected, might be compared. The 
degree to which these future criteria would extend the range of the summary 
performance measures could then be evaluated. The present study was 
.accomplished on only our specialist sampler general practitioners and 
medical faculty were not included; - 

Method and Results 

Before combining the 80 criterion scores for each physician, an- in- - 



spection of the criteria revealed that a high score on some of the criteria 
did not necessarily indicate a high quality of performance. For example, 
an excessive number of consultations requested by a physician would increase 
a total summed score but might reflect an undesirable method of practice./ 







■ " * ■ ‘ ' 

i'^This project was part of a larger project "Measurement and Predic ion 
of Physician Performance" supported by the .Department of Health. Education 
and VJelfare, contract No. PM-00017-()2 / , - 



2 



Table 1 

Title and Soinrce for Bach Variable for Uri>aii Specialist Sample 



Title 

1 Number of Time? Nominated as Outstanding Contributor by Urban Specialist Colleagues 

2 Number of Time: dominated as Preferred Consultant by Urban Specialist CoUeagues . . . . 

3. Number of Nomlr. ions as Outstanding Contributor or Preferred Consultant by Gener^ Ptactltlonsrs 

4. Number of Times Nominated as Outstanding Contributor by College of Medicine Faculty 

5. Rating of "Clinical Excellence" by Medical College Department Head 

6. Number of Different Residency Hospitals 

7. Number of Years Spent j'li Residency 

8. ‘udgcd Quality of National Board Certification 

9. Present College of Medicine Clinical Faculty Rank 

10. Mobility Rate in Professional Positions Since Receiving M. 

Total Number of Listings In Honorary Compendlums 

12. Gross Income from Medical Profession 

13. Number of Current Memberships In Scientific and Professional Societies 

14 . Average Judged Quality of Societies in Which Membership is Current 

15. Overall Occupational Satisfaction 

16. Number of Times During Career Invited to Serve as Editor of SclentUlc Or Professloui ‘aal 

1?! Number of Times During Career Invited to Serve on Scientific and Professional Advlso; rds 

is! Average Judged Quality oi Scientific and Professional Awards Received During Career . , . . . . 

19. Self-Reported Number of Contributions Made to Medicine 

20. Average Judged Quality of Self- Re ported Contributions Made tq Medicine 

21 ! Self-Reported Number of Non-Medical Contributions to Society 

22! Total Number of Papers Presented at Scientific and Professional Meetings During Career 

23. Average Number of Journal Publications Per 'Vear Since Recelvltig 

24. Average Level of Contribution to Publications; as Indicated by Senior vs. Junior Authorship Status 

25. Number of ResearchProjectswith Wliich Involved During Career 

25! Number of Scientific and Professional Journals Reviewed Regularly 

27 Nuniber of Subscriptions to Scientific and Professional Journals 

2s! Number of Articles in Scientific and Professional Journals Read In DecaU Each Month 

29. Average Number of Society Meetings Attended Annually 

30. Number of Postgraduate Courses Taken During Career 

31. Number of Refresher Courses Taken During Career 

32. Physician’s Evaluation of Usefullness of Drug Detail Men 

33. Extent of Physician’s Experimental Use of Drugs Provided by Drug Detail Men. .... . . - 

34! Number of Techniques Other Than Journals, Meetings, and Drug DetaU Men Used in Keeping Abreast 

35! Average Number of Formal Medical Consultations Called Into Monthly 

36. Average Number of Informal Medical Consultations Called Into Montlily 

37. Percentage of Patients on Which Consultations are Requested 

38. Number of Patients Seen Per Day 

39. Average Amount of Time Spent with Patients on First Visit 

4o! Average Amount of Time Spent in Explaining DUgnoses to Patients 

41. Proportion of Office Patients Treated Without Charge .. 

42. Proportion of Office Patients That Fall to Pay Physician fc r Services Rendered 

43. Self-Estimated Average Socioeconomic Level of Patients - 

44. Degree to Which Physician Adheres to Patient Appolntmeol Schedule 

45. Average Number of House Calls Made Per Week 

46. Degree to Which Physician Considers Psychological Factors in Diagnoses 

47. Average Number of Hours Per Week Devoted to Medical Practice 

48. Average Number of Hospitalized Patients 

49. Number of Hospitals in Which Physician Works 

50. Average Judged Quality of Hospitals in Which Physician Works 

51. Number of Hospitals’ in Which Physician Maintains Courtesy Priviledges 

52. Average Judged Quality of Hospitals In Which Physician Maintains Courtesy Priviledges • 

53! Number of Formal Responsibilities Physician has in Hospitals 

54. Average Judged Quality of Hospital Responsibilities 

55. Self-Estimated Value of Office Equipment 

56. Number of M.D. Assistants on Physician’s AncUlary Staff • 

57. Number of Nurses on Physician's Ancillary Staff 

58. Number of Technicians on Physician's Ancillary Staff * ' e-' ^ * 

59. Number of Clerical, Administrative, and Janitorial Workers on Physician s AncUlary Staff 

60. Average Number of Speeches on Medical Topics to Laymen Groups Per Year 

dl. Average Amount of Vacation Taken AnnuaUy 

62. Extent to Which Hiyslclan Flans and Maintains Leisure Time Activities 

63. Number of Current Memberships In Social and Avocatlonal Organization 

64. Number of Current Memberships In Civic and Political Organizations. 

65. Characteristics Physician Considers Important for Success In Medicine: Number of Common ^sponses. . . 

66. Characteristics Physician Considers Important for Success In Medicine: Number of Unco^on Responses. , 

67. Characteristics Physician Considers Important for Success In Medicine; "Commonness of Responses Ellclte 

68. Self-Rating of Success In Medicine 

69. Expert lynel Rating of OveraU Performance Baaed on All AvaUable Information 

70. Interviewer Rating of Condition of Riyslclan’s Office 

71. Interviewer Rating of 

72. Interviewer Rating of Physician’s Involvement In This Project 

73. Age at Which Received M. 1, 

74. Number of Years Between Receiving M. D. and Receiving National Board Certification 

75. Years of Experience Since Receiving M.D 

76. Individual Practice Rather Than Group, Clinic or Hospital Practice 

77. Hospital Practice Rather Than Individual, Group, or Clinic Practice 

78. Undergraduate Grade Point Average ^ 

79. Grade Point Average for First Two Years of Medical School • 

80. Grade Point Average for Last Two Years of Medical School 



Source 

Colleagues 
CoUeagues 
Interview 
.Interview 
ETepartment H“ad 
, Compendlums 
Compendlums 
. Expert Judges 
Official Records 
Interview and Official Records 
Compendlums 
Interview 
Interview 

Interview and Expert Judges 

Questionnaire 

Interview 

Interview 

. Interview and Expert Judges 
Interview 

. Interview and Exp>ert Judges 
, Interview 
Interview 

Interview and Compendlums 
.Interview and Compendlums 
Interview 
Interview 
Interview 
. Interview 
Interview 
Interview 
. Interview 
Interview 
, Interview 
Interview 
Interview 
Interview 
Interview 
Interview 
Interview 
. Intervlev/ 

Interview 
Interview 
Interview 
, Interview 
Interview 
. Interview 
Interview 
. Interview 
. Interview 

. Interview and Expert Judges 
. Interview 
. Interview 
. Int'' “View 

T' ' juages 

. Interview 
. Interview 
. Interview 
. Interview 
Interview 
. Interview 
. Interview 
Interview 
. Interview 
. Interview 
. Interview 
. Interview 
- Interview 
. Expert Judges 
, Project Interviewer 
, Project Interviewer 
, Project Interviewer 
Control Variables 
Control Variables 
. Control Variables 
Control Variables 
. Control Variables 
Official Transcripts 
. Official Transcripts 
, OMcial Transcripts 
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Therefore, items on which a high score was not necessarily indicative of 
superior performance vJere submitted to a panel of five physician judges who 
estimated the optimum level for each of these criteria. Scores above this 
estimated optimum were prorated downward, making them comparable to loiter 
scores on that criterion. Ten criteria were re -scored in this manner. 

Four different summary scores were derived for each of the 205 specialists 
studied by Price, et. al.., (1964). For the first three- suim^ ary scores this 
was acontplished by weighting each of the 80 criterion measures in three 
different ways: ( 1 ) Judged Weights— the 80 criteria were summed after each 

criterion had been assigned a weight based on its judged importance to 
medical practice. , These weights were assigned to the 80 criteria by having 
a panel of five physician judges rate the essentiality of each criterion 
for superior performance on a scale thap tanged from 5 ’’essential" to 

V \ ■ 1. 

1 "inconsequential"; ( 2 ) Equal Weights- -composite scores were obtained 
by simply summing the 80 unweighted criterion scores for each physician; 
and f 5) Statistical Weights— th^ weight for each criterion was derived by 
reversing the typical applicatir -f .1.. : tical technique o£ multiple 

correlation, using tlie 80 performance variables as ’.’predietprs" and the 

direct overall, ratrng otperform^ce (described in ,4 below) as the 

criterion. The bet -weights assigned to each variable in the regression . 

^ equation were need to weight the criteria. .Every summary score 

ment_oned above was achieved by weighting scores which had been previously 
standardized so that each criterion had essentially equal variability befo^ 
weighting. The > fourth score.' an overall ratin g fr om a crite j ron folde _ r 
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was obtained by having rdedical experts give an over-all rating of each 
specialist on the basis of his 80 performance criteria contained in his 



coded, anonyinous folder. A rating of 1-10 was assigned to each physician. 



with lov/er scores indicating superior perfoiinance. After the four suiranaiy 
scores were derived for each of the 205 physicians, these s^cores were 
converted to percentile ranks, as shown in Table 2. From that table it 
can be noted that physicians received very similar percentile rhnks for 
all foixr methods of score synthesis. This was true particularly at the high 
and low* ends of the continuum 



The next step in the present study was the securing of independent 
performance ratings on the physician sample, obtained independently ‘ of the 
data in -each criterion folder. These appraisals were made by five knowledge- 
able physicians who were selected on the basis of having had a wide ex- 
posure to a large number of Utah physici^s , Nambs of the 205 specialisits 
in the sample "under consideration were typed oh 3. xe 5 car^, -orie name to 
a card. After considering the quality" of each physician's i total pbrfor- * 
indncey the jiicjges sortefd ,ea^^ name on a^ nine -point scale which ranged' 'from 

.i'" ■ *’ 

; , ■ ■ , ' _ 

1 "outstanding" to 9 ’’extremely poor." A tenth dategoryj "I cannot fate ' 

this physician" was also included. The five .physician judges working ' • 

independently of any knowledge of the previously col leered criteria and 



their ratings vjef 6 averaged. Golumh Five in Table 2 gives the percentile 
ranks defived from these average "ratings. CScofes based on previously 



collected efitefih are in columns '1-4) , ' ‘ / 

!Ih'e’ 'finah -step in. the analysis o^-the five o^ef-all performance /■' 
measures consisted of intercorrelating these five variables. Table 3 



gives the results of this intc rebrre lation . It can be seen that the 
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Table 2 

• FIVE OVERALL MEASURES OF PHYSICIAN PERFORMAT'ICE 

* The first 3 columns below rank 205 specialists on 80 criterion measures of physician per- 
.forraance gathered to date. Each of these columns simply represents a different way of com- 
bining or synthesizing the 80 measures into one overall criterion of performance. Columns 
4 and 5 are ratings assigned by physician judges. Each row represents the relative position 
of one physician, as ranked by the methods described briefly above and in more detail below: 

1. Judged Weight . Measures were weighted according to essentiality for superior 
practice by 5 physicians and summed. 

2. Equal Weight . All 80 performance measures were given an equal weight before siimmation. 

3. Statistical Weight . Before summation, each measure we weighted by a multiple 
correlation which determined the relationship of each criterion to a global estimate 
of performance (see 4 below) . 

4. Overall rating of anonymous criterion folder . A 1-10 rating of overall performance: 
was malde by a panel of physicians who considered all 80 measures of each physician. 

5. Overall rating of physician by name . A measure of overall performance independent of 
our criteria was obtained by having five physician judges rate each specialist on a 
1-9 scale. A mean rating was then calculated for each specialist. 

The above ratings and rankings were subsequently converted into the percentile ranks shown below: 
, PERCENTILE RANKS PERCENTILE RANKS 



1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


9b 


99 


99 


95 


99 


76 


80 


69 


78 


48 


99 


98 


99 


95 


48 


76 


81 


80 


20 


59 


98 


99 


96 


95 


97 


75 


76 


56 


20 


72 


98 


97 


98 


99 


99 


75 


82 


69 


69 


62 


98 


98 


97 


95 


93 


75 


78 


45 


20 


72 


97 


97 


97 


95 


95 


75 


86 


80 


78 


91 


96 


95 


98 


95 


98 


75 


68 


83 


48 


25 


95 


96 


95 


95 


98. 


73 


74 


75 


08' 


67 


95 


95 


94 


91 


67 


73 


70 


85 


08 


82 


94 


94 


96 


95 


97 


72 


74 


73 


20 


86 


94 


94 


93 


20 


88 


72 


80 


74 


08 


59 


94 


93 


93 


91 


82 


71 


79 


90 


03 


72 


93 


94 


95 


20 


93 


,71 


61 


61 


48 


70 


93 




88 


78 


89 


71 


73 


91 


20 


48 


92 


^0 


84 


48 


39 


70 


68 


64 


48 


62 


92 


90 


87 


78 


23 


69 


62 


62 


08 


08 


. 91 


93 


90 


9l 


92 


69 


84 


67 


08 


67 


90 


92 


91 


95 


78 


68 


74 


71 


48 


82 


90 


86 


84 


95 


98 


67 


71 


71 


78 


10 


89 


89 


79 


95 


78 


67 


74 


76 


48 


45 


89 


90 


94 


20 


82 


66 


66 


61 


78 


76 


88 


85 


68 


48 


59 


65 


61 


83 


08 


62 


* 88 


87 


63 


08 


38 


65 


51 


16 




62 


88 


88 


53 


48 


86 


65 


59 


77 


48 


82 


87' 


88 


89 


48 


70 


65 


67 


78 


20 


76 


86 


83 


86 


78 


62 


64 


59 


38 


48 


23 


86 


81 


78, 


78 


48 


64 


71 


45 


20 


78 


86 


83 


77 


95 


84 


64 


61 


86 


78 


72 


85 


91 


89 


78 


89 


64 


56 


24 


20 


16 


85 


88. 


82 


20 


01 


64 


62 


85 


03 


86 


85 


82 


70 


20 


25. 


63 


71 


57 


20 


83 


84 


78 


66 


08 


95 


63 


63 


56 


08 


78 


84 ' 


85 


87 


48 


16 


62 


69 


60 


08 


93 


83 


77 


64- 


48 


23 


62 


71 


67 


48 


-82 


82 


84 


65 


20 


16 


61 


55 


46 


20 


89 


81 


73 


92 


01 


78 


61 


39 


15 


20 


62 


80 


85 


68. 


20 


67 


61 


68 


52 


20 


02 


80 


78 - 


79 


03 


76 


" 60 


69 


50 


48 


28 


. 79 


86 " 


92 


20 


76 


60 


60 


51 


20 


32 


• 79 


83 


62 


^ 03 
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60 


51 


51 


48’ 


' 82; 


78 


75 


74- 


- 48 


48 


59 


55 


57 


78 


48, 


. 77 


71 


86 


48 


67 


58 


57 


34 


20 


65 ^ 


O 76 


82 


81 


78 


48 


57 


48 


55 


48 


95 
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PERCENTILE 


RANKS 




/ 


PERCENTILE 


RANKS 




1 


2 


3 


4- 


5 


1 


2 


3 


4 • 


5 


57 


39 


13 


20 


32 


27 


34 


20 


48 


23 


57 


63 


49 


48 


91 


27 


22 


31 


91 


02 


57 


67 


75 . 


48 


62 


27 


32 


65 


48 


67 


57 


48 


42 


48 


32 


25 


25 


14 


78 


32 


56 


46 


18 


48 


48 


* 25 


32 


22 


48 


25 


55 


47 


81 


48 


05 


24 


42 


32 


48 


48 


55 


58 


72 


20 


76 


23 


35 


44 


48 


.32 


55 


31 


66 


91 


06 


23 


19 


29 


. • .48 . 


48 


54 


64. 


77 


48 


48 


23 


33 


60 


20 


16 


53 


47 


30 


, 48 


59 


22 


22 


22 


78 


62 


53 


.. 45 


63 


08 


57 


22 


30 


27 


48 


32 


51 


57 


35 


48 


86 


22 


25 


08 


20 


72 


50 


52. 


41. 


20 


. 89 


20 


37 


33 


78 


. 52 


49 . 


42 


72 


01 


57 


20 


21 


43 


48 


35 


49 


42 


53 


48 


72 


20 


21 


19 


78 


05 


49 


51 


41 


20 


38 


20 


22 


49 


78 


16 


48 
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38 


03 


48 


19 


31 


29 


48 


48 


48 


55 
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03 
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16 


36 


48 
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• 48 - - 
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40 


78 
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18 


11 


48 


78 


23 


47 


52 


47 


48 


67 


18 


25 


23 


48 


32 


47 


42 


35 


48 


16 


17 


29 
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78 


48 


46 


46 


47 


48 


48 


17 


17 


17 


78 


48 


46 


38 


26 


78 


01 


17 


19 


57 


48 


16 


45 


30 


55 
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16 


17 


19 


40 


48 


67 


45 


30 


21 


.20 


48 


. 17 
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59 
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44 
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.54 
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15 
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30 
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08 


43 
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.16 
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48 


32 


43 


47 
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Table 3 

Intercorrelations of Five Over-all Perfc -^.ance Measures 
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thv66 scores which represented different methods of criterion score 
weighting- -judged, equal, and statistical weights (variables 1, 2, and 
3 in Table 3)— were highly related to one another. Variable 4, the rating 
of the coded, anonymous criterion folders, and variable S, the independent 
appraisal of each physician by name, demonstrated sorae\i7hat lower relation- 
ships with the first three combined criterion scores and with each other. 
Nevertheless, these correlations were still significant far beyond the 
.01 level. It is challenging to note that the judges who looked through 
the criterion folders to make tlieir judgments must have given quite 
"different subjective ..aights" to nary of the criterion scores that were 
used in the first three exp licit methods for weighting and combining 

V 

these 80 types of criterion data'. . 

The ratings of specialists by name obtained' from the five physician 
judges,' were necessarily subjective and global in nature. They did, 
however, provide an estimate of how closely the criterion summary scores, 
based on 80 previously collected criteria, reflected actual performance. 

Our results ' indicate that the conposite criterion scores hold more promise 
than many critics have thought as reasonably accurate estimates of on-the-job 
performance, for use until they can be further extended and refined. If 

only the first three methods had been used, the weighting problem might 

I ' 

have been seen as a fairly trivial problem, but inclusions of the. last two ^ 
overall subjective rating methods brought out some important differences in 
computing weightingof the 80 separate criteria in the search for consistency 
on a composite overall rating. 
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An Examination of flultiple Approaches to Weighting the Performance Criteria 

Five methods of ranking physicians have already been discussed. Three 
of those methods (judged, equal, and statistical v/eights) were the results 
of weighting each criterion variable according to a specific weighting sys- 
tem. However, these three sets of weights represent only a few of the ways 
in whidi the criteria could have been weighted. The purpose of the next 
phase of the present study ivas to explore additional ways of assigning 

weights to the criteria. 

The \Corre latiohal Approach to VJeighting 

With the data available, it v;as possible to generate five sets of 
'‘correlational” v^eights. This procedure simply involved correlating each 
of the 80 criterion measures with one of the fivd overall ; summary scores 
mentioned iri the first part of this paper. Thus one set of correlational 
weights was obtained by correlating each criterion vjith the judged weight 
summary, score, the higher the 'correlati oh, the greater the weight. In the 

same manner, a' set of correlatiopal weights was calculated by correlating 
each critenon with the other four' summary scores, namely, the equally 
■weighted-summary score, the statistically weighted summarj^ score, the card 
ratings of" special^^^^^^ and the criterion folder rating. 


. ■' :- v 
- ■ 


; Other. .Weighting ! Approaches \ . , y 

.1 .i^^o 6thet methods of weighting have already been discus . 


/yv., t- 


earlier section of ythls; paper . med in determriing / 
the frist- summdt^ dsed to rahk^i*ysician Table 2. The statistical 

apprr,:ch i .e . , weighting by m.ultiple regression, was also used^ in weighting 
the 80 criteria, and was obtained by a regression of the 80 criteria against 


ERIC 
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the criterion folder score. The last set of statistical weights was cal- 
culated by multiple regression techniques, this time against the overall 
rating of physicians by name. 

Thus, the eight different sets of weights listed below were obtained 
(equal weights were not used in this analysis since every criterion measure 
received a v/eight of 1 in this method) . 



Eight Sets of IVeights 



General Method 
o£ VJeighting 


Weight Set # 


Specific Source 




1 


Correlation between each criterion and 
the equal weight summary score. 




2 


Correlation between each criterion and 
the judged weight summary score. 


I CORRELATIONAL 


3 


Correlation between each criterion and 
the best-statistical summary score. 


i V _ r. _ . . 


- . 4i 


Correlation between each criterion and 
the rating of the criterion folder. 




' • ■ . 5 

■ . ■ ’N ' 


Correlation between _each criterion and 
the rating of each specialist by hane. 


II JUDGMEOT 


, . . . 6 


Weight assigned by e.xpert judgment ^1, 
of the criteria in terms of importance . 



■ 7 . Beta' weights : from regression, of 80 . 

> ' criteria on the criteil on folder rating. 

Ill BEST STATISTICAL - ^ _ ' , / . ' _ 

8 ' . ! Beta weights from regression of 80 >; 

criteria on the rating of specialists by 
* ■ . _ n&in© p. _ . / 




- 11 - 



Tables 4 and 5 present an analysis of the 80 criteria in terms of the 
eight sets of weights. For clarity of presentation and as a better indication 
of overall standings of the criteria, the eight sets of v;eights were summed 
to determine a grand total importance level for each criterion. In both 
Tables 4 and 5, the 80 criteria are divided into three levels of importance. 

It was also of interest to note how much variability there was in the 
weights assigned to each criterion- This .provided an indication of the 
stability (consistency) or the instahilit} of Lr 3 performance measures. 

TabT 4 presents the criteria which v^ere falzrly- stable and consisten". 

Thes » measures tended to receive the same weig^' ts regardless of whicn of 
the o methods was used to assign weights. Tab^e 5 presents criteria found 
to be rather unstable or inconsistent, receiving comparatively different 
weights across the 8 methods. 

An examination of Tables 4 and 5 provides some idea, of the present 
status of our criterion analysis. In Table 4, some of our 80 criterion 
measures are both quite stable and important, while others tend to be 
stable and unimportant, not really worth a great deal of further consider- 
ation. These two extreme instances are fairly clear cut. In Table 5, 
however, many of our criteria are less clearly delineated. That is, they 
may be given considerable weight or importance by some weighting methods, 
but little by others. These discrepancies^ may give us some insight into 
the weights themselves, b^ also proyide some real problems in deciding 
which : criteria in the top two levels of importance are worthy of further 

consideration- and refinement. ^ 

r . - "" ■ V-' ■ .. ■/ : , ■ ' . '‘' 
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TABLE 4 



Criteria Receiving Highly Similar Weights by Eight Methods of Weight Assignment 
(Criteria are Arranged in Rank Order of Importance) 



CONSISTENT CRITERTA--.OF GREAT IMPOllTANtE 




Number of times nominated as outstanding contributor by colleague.': x 

Number of times nominated as preferred consultant by colleagues x 

Number of times nominated as preferred consultant by General Practitioners x 

Judged quality of national board certification x 

Total number of listings in honorary compendiums x 

Self-reported number of contributions made to medicine x 

Number of formal responsibilities physician has in hospitals x 

Expert panel rating of overall performance based on all available ±nfo3iaticn x 

Grade-poin'^ average for first two years of medical school x 



CONSISTENT CRITERIA— OF MODERATE IMPORTA'fE 



Grade-point average for last two years of medical school 

Number of current memberships’ in Civic and political organizations 

Average number of house calls made per, week. 

Interviewer rating of likeability 
Years of experience since receiving M.D, 

Number of hospitals in which physician maintains courtesy privilege^::^ 

Average number of speeches on medical topics to laymen groups per year 
Interviewer rating of physicia:n*s involvement in this project 
Extent of physician's experimental use of drugs provided by drug derail men 
Proportion of office patients that fail to pay physician for services rendered 

CONSISTENT CRITERIA— OF LITTLE IMPORT/^CE 

Mobility rate in professional positions since receiving M,D* 

Self-reported number of non-medical, contributions to society 
Self-estimated average socioeconomic level of patients 
^elf-estimated value of office equipment 

Number of M.D, assistants on ^^ysician's ancillafy staff ' 

Number of technicians on ph>^sicia.n*s ancillary staff 

Characteristics considered important for success: number of "uncoimnon'l responses 

Number of patients seen per day j /, ' 

Propartibn of office patients treated without charge 

Judged quality of^hospitals in which physician'^maintains courtejsy privileges 
Characteristics considered important for success: number of "common" responses ^ 
Characteristics considered important for succe«?s: "commonness" of responses ^ 
Self-rating of success in medidine , . j . ' 

Percentage of patients oh which consultations are, requested 
Physician's evaluation of usefulness of drug detail men, ' 

Number of hospitals in which physician works j . ", 

Number of nurses on physician '.s ancillary^staff ^ 

Extent to which physician plans and maintains leisure" time activities 
Individual practice rather than group > clinic of hospital practic^ 

Number of different residency hospitals ^ 

Number of clerical ^ administrat:^oh, and janitorial workers on staff 
Age at which received M*D, 

Number of years between’ receiving M.D. and receiving national boi 2 .ra rerti:ricatlon 
Hospital pra.ctice rather than individual j group, or clinic, practice. 



"pnV^^Judg = Criterion, dependent primarily upon judgment 

nbj s Criterion gathered from primarily "objective" sources’ 
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x 



X 

X 

X 



X 

X 

X 

X 

X 

X 




Criteria I'leijjhted Differently by Each of the Eight Weighting Methods 
f Criteria Are Arranged in" Rank 'Order of 'Ir^portance) 



LR^STARLE CRITEPIA--OF GREAT I^TORTATICE 

Murnber of tinges nominated as outstanding contributor by college of medicine facLf ty 
Present college of inedicine clinical faculty rank 

Average number of journal publications per year since receiving M,D. 

Average judged quality of societies in wl'ich membership is current ^ ^ 

y'verage judged aualitv of scientific and professional awards received during carec . 

Total number of paners presented at scientific and prof, meetings during career 

Rating of "clinical excellence" by medical college department head 

Number of scientific and professional journals reviewed regularly ^ 

huinber of times during career-invited to serve as editor of sclenti-ic journal 

Average judged quality of self-reported contributions made to medicine 

Times "during career invited to serve as editor of scientific journal 

Number of subscriptions to scientific and professional journals 

Average number of xcrmal medical consultations called into monthly 

Contribution to publications , as indicated by sr. vs. jr, author status 

Number of research projects with which involved during career 

Intervievjer rating of physician's involvement in this project 

UNSTABLE CRITERIA--OF MODERATE INTORTANCE 



Judg, Obj . ' 



X 

X 

X 

X 



X 

X 

X 

X 

X 

X 

X 



Gross income from medical profession v, „ „«.t. 

Number of articles in professional journals read in detail eac_. mon 
Average judged quality of hospital responsibi lities ^ ^ 

Average amount of times spent with patients on first gisit 
Av6Tap[e numbsr of hoS'oi'ta.lizGd pa.'tion'ts 
Number of years spent in residency 

Untisual techniques of keep abreast , - 

Average number of informal medical consultations called into month y 

Average judged quality of hospitals in which physician^wprics ^ 

Number of cuirent. memberships in social and avocational organiza 

Undergraduate grade point average ^ 

Average amount of time spent in ejqjlaining diagnoses to patients 

Average amount of vacation taken annually 

Average number of society meetings attended annually 
Number of postgraduate courses taken during career : 

imigTARTF rRTTERIA— OF LITTLE IfTOPTANCE^ 

Overall occupational satisfaction - 

Number 'of refresher courses tak during career ^ crr-Wo/intp 

Degree to which physician adheres to 

Degree to which physician considers psychological factors in 

Number of current memberships in scientificand professional societies 

Average number of hours per week devoted to medical practice 



*Judg. 
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cxiterion dependent ■’primarily op6n judgmept ^ 
'dt'ft'e'iabn gathere primarily -^’’objective - sources 



X 

X 



X 

X 

X 

X 

X 



X ^ 
X 

X - 



X 

X 

X 

X 

X 
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Sumnary and Conclusions 



The first phase of the present study examined three methods of syn- 
thesizing and combining. 80 criteria of physician performance into single 
composite scores. In addition, an independent rating of the performance 
of each of the 205 specialists in the sample was obtained as vjas a rating 
of a coded anonymous criterion folder. Thus, five overall scores were 
obtained for each physician. These scores were then arranged in rank order 
in profile form for eadi specialist to indicate the relative standing of 
each subject on Sll five scores. The three overall scores based on the 
differential weightings of the 80 performance measures were highly correlated 
with each other, and moderately correlated with the other two independently 
obtained overall scores. 

The second ohase of the study dealt with how the 80 criteria mentioned 
above were weighted by three distinct techniques including judgmental i 
correlational, and best-statistical methods of weight assignment. The 
criteria we re subsequently classified in terms of their importance and 
variability under all weighting methcdsC / 

The major conclusions of the study were: (1) The 80 performance criteria 
yielded ^remark ab ly simi lar summaiy scores when summed and weighted by three 
different techniques. (2) Substantial relationships we-re demonstrated 
between the stnnmary scores based on, the criteria per se and the independent 
appraisals "6f physician performance. (3) Many of the criteria examined i 

in this study vre re extremely vafiab le ; over different p^eightiTig methods '' 
while others vjrere quite _ stable, an indication of the complexity of the whole 
weighting problem and the entire problem pf getting one best o'O'erall score. 



